Case study
iTinker
One live, searchable catalog for second-hand products scattered across the web.
- Role
- Full-stack Engineer
- Timeline
- Mar 2023 to Jul 2024
- Location
- Chandigarh, India
- Stack
- Spring Boot, Kotlin, Next.js, PostgreSQL, Elasticsearch, Kafka, Server-Sent Events, WebSockets, Jsoup, Selenium, Redis, AWS Elastic Beanstalk, Amazon S3
- Faster responses from the scraping engine
- 4×
01
Context
Product discovery for second-hand goods, across marketplaces that were never built to work together.
iTinker pulls second-hand listings from many separate marketplaces into one place, where people can search them and follow them as they change.
As a full-stack engineer I worked on its microservices, search, real-time updates, the scraping engine and the AWS deployment.
Discovery pipeline
Many sourcesFragmented marketplaces
One engineJsoup · Selenium · Redis
One indexElasticsearch, live via SSE
02
Challenge
Fresh data from slow, messy sources, without ever making users wait for it.
- Fragmented sources
- Every marketplace structures its listings differently. Some pages parse straight from HTML. Others need a real browser to render before there’s anything to read.
- Freshness
- Second-hand listings change fast. Discovery is only useful while the data is current.
- Speed
- Collecting from other sites is slow by nature. Users shouldn’t feel it.
- One catalog
- Listings from different sources had to be searchable together, as if they came from one place.
03
Approach
Collect smart, cache hard, index once, and push changes the moment they happen.
- Split by responsibility. Kotlin and Spring Boot microservices behind REST APIs.
- Match the tool to the source. Jsoup where the HTML can be read directly. Selenium only where a real browser is required.
- Cache aggressively. Redis caching inside the scraping engine, so repeated work is skipped instead of sped up.
- Search in a search engine. Elasticsearch for discovery, PostgreSQL for everything that has to persist.
- Push, don’t poll. Kafka moves updates through the system. Server-Sent Events and WebSockets carry them to the browser.
How a listing travels (simplified)
- CollectJsoup · Selenium
- CacheRedis
- StreamKafka
- IndexElasticsearch
- DeliverSSE · WebSockets
04
Implementation
What I built.
| Area | Work |
|---|---|
| Services | Microservices in Kotlin and Spring Boot, exposed through REST APIs |
| Scraping | A scraping engine that combines Jsoup, Selenium and Redis caching |
| Search | Product search on Elasticsearch, with PostgreSQL as the source of truth |
| Real-time | Live product updates through Kafka, Server-Sent Events and WebSockets |
| Frontend | The Next.js product interface |
| Infrastructure | Deployed on AWS Elastic Beanstalk, with storage on Amazon S3 |
05
Result
Responses got 4× faster.
Bringing Jsoup, Selenium and Redis caching together in one scraping engine made responses four times faster. Kafka, SSE and WebSockets kept product updates flowing to users as they happened.
Relative response time
06
Engineering lessons
- Caching is architecture
- In a product built on scraping, the work you skip matters as much as the work you speed up.
- Real-time is a pipeline
- Freshness depends on every stage: collection, streaming, indexing and delivery. The slowest one sets the pace.
- Use the lightest tool that works
- A full browser is powerful and expensive. Reach for it only when plain HTML parsing can’t do the job.
- Search deserves its own store
- A dedicated search index keeps discovery fast without bending the main database out of shape.