Should I worry if I used clove/peppermint oil flea spray on my cat?

Clove and peppermint only annoy your cats' and dogs' sence of smell. Since they have a much more keen sense of smell these aromas are overpowering to them. Fleas do not get repelled by peppermint and clove. I would not use it near my pets.
The best way to get rid of fleas is as follows:
Unfortunately fleas only hop on the body of your pet when they want to feed, then they hop off again and stay in your pet's environment. So you need to treat all pets in the household as well as the environment (which contains the fleas, their eggs and larvae).
Flea treatment: 
1) Get some flea medication such as a Flea spray or a spot on from a pet shop or your veterinarian and apply monthly 2) get flea powder, put some in your vacuum cleaner bag and vacuum your house and furniture and also non-carpeted areas frequently to remove flea larvae and eggs which live off house dust., Dispose of the bag in the trash outside. Hot-wash all the pet's bedding,
If you do this diligently for a few months you will get rid of the fleas.
Flea products:
Spot-ons and/or chews
Advantage, Advantix
Advantage Multi (Advocate in the UK)
Frontline (ticks as well)
Bravecto spot on (ticks as well) - for cats, in tablet form for dogs
Simparica, Nexgard
Revolution (stronghold in the UK)
Tablets: comfortis (will only kill fleas once)

Collar: Seresto (ticks as well)

Indorex:
The active ingredients of the product are Permethrin, Pyripoxyfen and Piperonyl Butoxide. Between them, these three substances kill adult fleas and stop the development of eggs and larvae of fleas and house dust mites.
However, as permethrin is toxic to cats, I would not use it regularly and certainly would not leave the cats in the house while spraying it.