'%20fill='%23000000'%20stroke='none'%3e%3cpath%20d='M%201784.53%205631.9%20C%201775.53%205618.9%201692.53%205479.9%201598.53%205322.9%20C%201395.53%204983.9%201355.53%204919.9%201342.53%204915.9%20C%201332.53%204912.9%201315.53%204938.9%201090.53%205311.9%20C%201023.53%205423.9%20960.53%205523.9%20951.53%205534.9%20C%20936.53%205549.9%20918.53%205552.9%20816.53%205552.9%20C%20751.53%205552.9%20696.53%205548.9%20692.53%205542.9%20C%20689.53%205537.9%20721.53%205475.9%20763.53%205405.9%20C%20806.53%205335.9%20855.53%205252.9%20872.53%205222.9%20C%20889.53%205192.9%20941.53%205106.9%20987.53%205032.9%20C%201032.53%204958.9%201161.53%204746.9%201272.53%204562.9%20C%201383.53%204378.9%201522.53%204147.9%201581.53%204050.9%20L%201689.53%203872.9%20L%201823.53%203872.9%20C%201897.53%203872.9%201957.53%203874.9%201957.53%203876.9%20C%201957.53%203879.9%201887.53%203997.9%201800.53%204139.9%20C%201513.53%204612.9%201487.53%204657.9%201487.53%204674.9%20C%201487.53%204697.9%201788.53%205192.9%201802.53%205192.9%20C%201807.53%205192.9%201830.53%205162.9%201852.53%205125.9%20C%201874.53%205088.9%201902.53%205042.9%201913.53%205023.9%20C%201925.53%205003.9%201993.53%204890.9%202065.53%204772.9%20C%202395.53%204228.9%202458.53%204121.9%202452.53%204112.9%20C%202448.53%204106.9%202382.53%204102.9%202287.53%204102.9%20C%202200.53%204102.9%202126.53%204098.9%202123.53%204094.9%20C%202120.53%204089.9%202147.53%204038.9%202183.53%203979.9%20L%202249.53%203872.9%20L%202563.53%203872.9%20C%202736.53%203872.9%202877.53%203875.9%202877.53%203878.9%20C%202877.53%203882.9%202812.53%203991.9%202734.53%204121.9%20C%202537.53%204445.9%202057.53%205239.9%201929.53%205452.9%20C%201801.53%205664.9%201804.53%205662.9%201784.53%205631.9%20Z'%20style='fill:%20rgb(215,%2032,%2038);'/%3e%3cpath%20d='M%202191.53%205539.9%20C%202187.53%205527.9%202443.53%205099.9%203052.53%204100.9%20L%203191.53%203872.9%20L%203324.53%203872.9%20C%203397.53%203872.9%203457.53%203874.9%203457.53%203876.9%20C%203457.53%203879.9%203366.53%204031.9%203255.53%204214.9%20C%203144.53%204397.9%202964.53%204696.9%202854.53%204877.9%20C%202744.53%205059.9%202607.53%205285.9%202550.53%205380.9%20L%202445.53%205552.9%20L%202321.53%205552.9%20C%202226.53%205552.9%202195.53%205549.9%202191.53%205539.9%20Z'%20style='fill:%20rgb(215,%2032,%2038);'/%3e%3cpath%20d='M%201029.53%204386.9%20C%201014.53%204360.9%20993.53%204325.9%20982.53%204308.9%20C%20970.53%204291.9%20938.53%204237.9%20909.53%204187.9%20C%20880.53%204138.9%20829.53%204053.9%20797.53%203999.9%20C%20764.53%203944.9%20737.53%203894.9%20737.53%203886.9%20C%20737.53%203874.9%20759.53%203872.9%20864.53%203874.9%20L%20991.53%203877.9%20L%201099.53%204049.9%20L%201207.53%204220.9%20L%201144.53%204326.9%20C%201110.53%204384.9%201076.53%204432.9%201069.53%204432.9%20C%201062.53%204432.9%201044.53%204411.9%201029.53%204386.9%20Z'%20style='fill:%20rgb(215,%2032,%2038);'/%3e%3c/g%3e%3c/svg%3e)
Farmland Market: Property Feed Ingestion & Data Normalisation
Project Overview
Farmland Market, a property marketplace focused on farmland and rural estates, needed a reliable system to ingest and update property listings from external sources. The platform's 'Hive' import function handles rich data—descriptions, images, broker details, geolocation, and key features—but was experiencing reliability issues and data quality inconsistencies. Scalexa delivered a robust ingestion pipeline with improved detection logic, controlled batch sizes, and a structured roadmap for continued platform improvements.
Client and Product Context
Farmland Market serves buyers and sellers in the rural property sector, aggregating listings from multiple external sources including Rightmove. The platform relies on automated imports to keep listings current, with each property containing rich metadata: acreage, location coordinates, agent information, images, and detailed descriptions. The engineering challenge was to make this pipeline reliable, maintainable, and scalable as the marketplace grows.
The Challenge
The existing import process faced several interconnected issues that required systematic resolution:
- Import jobs were 'falling over'—long-running processes exceeded PHP max_execution_time limits
- Acreage detection logic failed on variations: 'acre', 'Acre', 'ACRES', 'acres', and similar permutations
- Description formatting inconsistencies: paragraph spacing varied depending on broker-authored content
- High import volume made testing slow and debugging difficult
- Upstream image fetch failures (Rightmove 404s) needed graceful handling
- No structured approach to prioritising follow-on improvements
What We Did
We approached this as a reliability engineering project, focusing on measurable improvements to import stability and data quality. Our work followed a pragmatic, step-by-step methodology:
- Diagnosed root cause of import failures—PHP execution time limits on large batch imports
- Implemented acreage threshold filters (initially 100 acres, later 300 acres) to control import volume during testing
- Improved area detection logic to handle all common 'acre' permutations in listing data
- Configured scheduled imports to run hourly, allowing incremental progress without constant restarts
- Normalised description handling from feed to database, with clear documentation on formatting limitations
- Identified and documented image fetch failures, implementing resilient ingestion with appropriate error handling
- Structured a prioritised backlog of mini-projects for continued development
Solution Architecture
The ingestion system comprises several interconnected components working together to deliver reliable property data:
- Ingestion/Import Job: PHP-based scheduled task handling feed parsing, data transformation, and database writes
- Database Layer: Normalised property records with proper indexing for acreage, location, and listing metadata
- Admin Tools: Interface for monitoring import status, reviewing failed records, and manual data correction
- Scheduler/Cron: Hourly execution ensuring incremental progress and reduced per-run execution time
- Proxy/File Service: Asset serving layer for property images with fallback handling for missing upstream assets
- Source Connectors: Configurable adapters for external feeds (Rightmove and future sources)
Technical Highlights
Several engineering decisions were critical to achieving stable, maintainable imports:
- Import throttling via acreage filters: Reduced batch sizes from thousands to under 50 properties during testing phases
- Execution-time tuning: Increased max_execution_time combined with smaller batches prevented timeout failures
- Incremental scheduled runs: Hourly imports accumulate progress rather than attempting full reprocessing each run
- Robust acre parsing: Regular expressions handling 'acre', 'Acre', 'ACRES', 'acres', 'ac', and edge cases
- Upstream formatting variance: Clear documentation distinguishing fixable issues from broker-authored content requiring NLP for correction
- Resilient image ingestion: Graceful handling of Rightmove 404 responses with logging and retry logic
Security and Reliability Considerations
Enterprise-grade reliability requires attention to operational concerns beyond core functionality:
- Rate limiting on external API calls to respect upstream service constraints
- Input validation on all ingested data before database insertion
- Comprehensive logging for import runs, including success counts, failures, and skipped records
- Idempotent imports: Re-running the same batch produces consistent results without duplicates
- Retry logic with exponential backoff for transient failures
- Monitoring hooks for alerting on import failures or anomalies
What's Next
With the core pipeline stabilised, a structured roadmap guides continued platform improvements:
- Query controls in Voyager: Adjust upstream queries (Datafiniti) for more targeted data retrieval
- Proxy/file service enhancements: Improved caching and fallback handling for listing assets
- Feed services: Consume listings from additional URL-based sources
- Agent fields: Enhanced broker/agent metadata capture and display
- Exclusion logic: Configurable rules to filter unwanted listings at import time
- Geolocation updates: Improved coordinate accuracy and postcode-based fallbacks
Outcomes
- Eliminated import failures—stable end-to-end imports with no timeout errors
- Controlled batch sizes under 50 properties during testing for rapid iteration
- 389 properties successfully imported and maintained in production
- Acreage detection now handles all common variations accurately
- Improved admin workflow for reviewing and editing imported listings
- Structured backlog providing clear roadmap for continued development
Why It Matters
For a property marketplace, reliable data is the product. By stabilising the import pipeline, improving data quality, and establishing a pragmatic roadmap, Farmland Market now has the foundation for sustainable growth. The platform can confidently scale its listing volume while maintaining the data accuracy that buyers and sellers depend on. Scalexa's experience—including work for brands like Walmart, Coca-Cola, ESPN, UNICEF, and FIFA—ensures enterprise-grade engineering practices applied to startups and SMBs alike.
Services Behind This Project
The same senior teams that delivered this work:
"I've worked with Scalexa for over 5 years. They provide a great service. Would recommend."