What service is implementing your security groups? Nova or Neutron?? If its neutron you will need to set the quota there.

root@controller1 ~(keystone_admin)]# neutron quota-show
+---------------------+-------+
| Field               | Value |
+---------------------+-------+
| floatingip          | 50    |
| network             | 10    |
| port                | 50    |
| router              | 10    |
| security_group      | 10    |
| security_group_rule | 100   |
| subnet              | 10    |
+---------------------+-------+

[root@controller1 ~(keystone_admin)]# neutron help quota-update
usage: neutron quota-update [-h] [-f {shell,table,value}] [-c COLUMN]
                            [--max-width <integer>] [--prefix PREFIX]
                            [--request-format {json,xml}]
                            [--tenant-id tenant-id] [--network networks]
                            [--subnet subnets] [--port ports]
                            [--router routers] [--floatingip floatingips]
                            [--security-group security_groups]
                            [--security-group-rule security_group_rules]
                            [--vip vips] [--pool pools] [--member members]
                            [--health-monitor health_monitors]

Define tenant's quotas not to use defaults.

optional arguments:
  -h, --help            show this help message and exit
  --request-format {json,xml}
                        The XML or JSON request format.
  --tenant-id tenant-id
                        The owner tenant ID.
  --network networks    The limit of networks.
  --subnet subnets      The limit of subnets.
  --port ports          The limit of ports.
  --router routers      The limit of routers.
  --floatingip floatingips
                        The limit of floating IPs.
  --security-group security_groups
                        The limit of security groups.
  --security-group-rule security_group_rules
                        The limit of security groups rules.
  --vip vips            The limit of vips.
  --pool pools          The limit of pools.
  --member members      The limit of pool members.
  --health-monitor health_monitors
                        The limit of health monitors.

output formatters:
  output formatter options

  -f {shell,table,value}, --format {shell,table,value}
                        the output format, defaults to table
  -c COLUMN, --column COLUMN
                        specify the column(s) to include, can be repeated

table formatter:
  --max-width <integer>
                        Maximum display width, 0 to disable

shell formatter:
  a format a UNIX shell can parse (variable="value")

  --prefix PREFIX       add a prefix to all variable names