The biggest risk with a flexible commission model is that nobody can explain it. If you can only describe it in a spreadsheet, your vendors cannot understand it either. Before launch, write the rules as three or four examples in plain words: a sale, the commission taken and the payout. Put those on the vendor page and check that a new vendor can follow them without asking you.
Test every rule with a real order. Set up a vendor, a product and a rule, place an order and check that the amounts in the WCFM screens match what you expect. Do this for each exception you plan to offer. A rule you have not tested is a rule you will be correcting by hand later, in front of an unhappy vendor.
If you sell in more than one currency, test the shopper side as well as the vendor side. Check what a shopper sees at checkout and what the vendor sees in their dashboard. Finally, keep the number of exceptions low. Every exception is a question a vendor will ask, and a question you will answer for as long as the marketplace runs.