Hi,
You say it is impossible to authenticate with curl / wget? I am sure that can be done. My last project was similar to yours, employer was have selenium script, which is really slow (generally all selenium scripts are slow), and I used simple http requests with ruby, made reverse engineering of js code, also cookies and sessions, and I made script much faster, from 2.5 min to 5 sec.
Yes, everything can be done. I can manage cookies within my scripts, and I can send cookies what need to be sent to server. I also act as normal user, as I follow every request as like with normal browser. I can use asynchronous calls, once I am authenticated. I can use proxies, etc.
I give you almost all details. But I need to know what website you want to scrape, so I can give you right price for this project.
I work with ruby only.
Best regards,
Sasa Sekaric