Periodically running a go web-scraper?

Question

I have created a web scraper that keeps an active database of craigslist postings in a certain category.

I want to run this every 10-20 minutes to periodically update the database but I don't know what services I have available to me to do this aside from heroku.

I've gotten my app built in heroku using vendor but from that point I've got no clue how to run it periodically.
Any resources for me?

Lex Hoffman · Accepted Answer

This seems like something that could be solved with a FaaS like AWS Lambda with a CloudWatch trigger. Have you looked into that?

Chris Gregori · Answer

Corn Seems like a good option but it also seems like I'd have to keep a heroku worker running constantly for it to work which is more money. I don't have an option to run this locally

Chris Gregori · Answer

It's a golang question, I'm open to any recommendations on what others are using to host their golang-based web services. What you suggested was cron which would require me constantly running a server as opposed to specifically opening a server to run the task then shut it down. Which, though I don't have a ton of experience sounds like it's going to cost more money whether I'm using what you're describing as a "real" server or not.

Go Language

Similar Discussions

Share Interesting projects built with Go?

Https Server with a single insecure/http endpoint? Share your thoughts.

Is this a bug in my Code?

On a mission to build Next-Gen Community Platform for Developers

Facebook

Twitter

Linkedin

Instagram