Just rate the quality of their product. They produce analysis. The quality of analysis is easily measured and quite universally done. It's almost assimple as the baeball player thing with the advent of modern computers. I always see how badly the analyst missed, my standard is the facts legally required to be reported. Skip the forecasting and prediction bullshit, that isn't even on the table any more as a reality. Base production and comprehensiveness and unique insights are easy to peg as well.
I see how it could be done, though I do not know if it can be implemented. For instance, let's say 3 analysts forecast for the upcoming quarter XYZ company's earnings: Analyst A: says it will be 23c/share Analyst B: " 21c/share Analyst C: " 14c/share Obviously, Analyst C is the outlier and is taking the most risk. Let's say XYZ reports 22c/share. As a result Analyst C gets fired. However, 1 year later the SEC investigates the firm's accounting and finds it had grossly overstated its earnings. Then the company is forced to restate its earnings to 13c/share for that quarter. Although Analyst C's analysis has been vindicated, nonetheless he is out of job. So, what will probably happen to prevent such consequences is that analysts will herd together with the company to come up with a "consensus" earnings estimate of 21c share, so that the company can beat it by a penny, and look good in the eyes of investors. If subsequently they are wrong as outlined above, they can simply plead "shock" that such outrageous behavior occured. But at least kept their jobs. IOW, A bird in hand is worth two in the bush. Now, how is that different from today's system?