Harnham’s Post

How was your first experience with data quality? Let me share my experience because many of you have probably done the same, or this path is ahead of you. My background is in software engineering, Java, .NET, backend, frontend, algorithmics, etc. Later, I moved to data engineering. My first promotion to a technical leader was 22 years ago. That is how long I have been leading teams and architecting software. I was assigned to design a data quality solution and organize a data quality team that would pick and configure data quality checks. It was a big data quality project for a big company. Up to 13 data quality engineers reported directly to me for four years. We were progressing deeper and deeper into the field of data quality, exploring new ways to identify data quality issues. The team under my command designed more than 1,000 custom data quality checks. Here are the challenges that I faced when building the team: 👉 Finding a data quality engineer with experience in the field is tough. 👉 The best way to build the experience in the team is to hire data scientists, even junior data scientists, and teach them about data quality. 👉 You must teach the new hires the meaning of every data quality dimension because the articles about data quality dimensions on the Internet all look the same. They provide a dictionary definition of the dimension without any examples of data quality checks that cover it. The last point is the hardest. Although there are many materials and articles about the importance of data quality, very little has been published with real examples of data quality checks. The same problem also affects business sponsors and data asset owners. You have to educate them and tell them which of their data quality requirements are impossible. What is your experience finding a list of data quality checks covering every data quality dimension? Are there any good references? #dataquality #datascience #dataengineering

📊 South East-QLD Software Engineering Jobs Report September 2024 Summary! 📫 Click on the "Visit my website" button under my profile pic and subscribe for free to receive monthly, quarterly, and FY email reports with access to the spreadsheets. #️⃣ of Jobs posted - 137 (90% Perm, 4% Fixed-Term, 5% Contract, 1% Part-Time). Top 5 role functions;  ▶ Back End - .Net/C# - 20% ▶ Lead - People (Team Lead, EM, Head of) - 15% ▶ (TIE) Back End - C/C++ and Back End - Python - 9% ▶ Back End - Node.js - 8% Top 5 Sectors;  ▶ Software Development/SaaS - 33% ▶ ITS/Consulting - 19% ▶ Financial Services - 16% ▶ (TIE) Retail/Hospo/Food & Beverage, and Government - 3% Other Insights;  ▶ 47% of jobs were posted by local HQ'd companies (17% uplift from Aug) vs 53% from interstate/overseas. GREAT TO SEE! ▶ We saw a B2B weekly decline in job numbers throughout Sep. Week 1 we saw 41 jobs posted, Week 2 - 36, Week 3 - 31 and Week 4 - 29. That’s a 29% slide. Typically Oct/Nov are strong recruitment months with companies wanting to secure resources by EOY. Let’s see how it plays out. The QLD State Election on the 26th, and the US Election on the 5/11 will have an impact. I know many projects have halted across government departments, and US investors might be more reserved about their spending.  ▶ We saw strong demand again for .Net/C# taking out the #1 spot with a 5% increase. The need for people leaders was apparent again, but we saw a 6% drop in technical leadership jobs, falling out of the top 5.  ▶ It’s interesting to see C/C++ again, driven mainly by companies across the Aviation & Aerospace, Defence and Transport/Logistics/Supply Chain sectors.  ▶ Node & Python made it on the list; I expect them both to be regular features. Node seems to be the go-to language on the back end for most software start-ups/scale-ups (Golang is gaining ground though), and Python a symptom of the big push towards AI/Data. NOTE; ▶ All data is derived solely from LinkedIn.  ▶ I do not include adverts from recruitment agencies.  ▶ Any & all feedback is welcomed 😃 #southeastqldsoftwarejobs #southeastqldsoftwareengineering #softwareengineers #softwaredevelopers #jobsdata

Come and join the Bennett Institute for Applied Data Science, at the University of Oxford, led by Professor Ben Goldacre! We built OpenSAFELY, a revolutionary open source platform for secure clinical research. We're hiring at all levels for software engineers and product managers. The blog post linked here gives you the facts. But to give you a flavour of what we think passes as a good time, we're talking things like: 🍩 Making meetings non-terrible 🐙 Extreme keyboard customisation 🍝 Formatting, linting, and building all our code consistently, wherever it is 🐦 Talking to our users 🪄 Automating All The Things with custom slack bots 🦄 Audaciously/foolishly designing new query languages 🍌 Puns 🎨 Emoji-driven metric collection 🚀 Making Freedom of Information requests 🕵️♂️ TFW generative testing finds obscure bugs in third party libraries 🐑 Knitting 🌈 Meeting IRL and doing (board games|music|esoteric talks|just chatting) 🔍 Looking everywhere for more excuses to use sqlite 💃 Dance interpretations of quicksort (not by us...) Let nobody say we don't know how to have fun.

📊 South East-QLD Software Engineering Jobs Report September 2024 Summary! 📫 Click on the "Visit my website" button under my profile pic and subscribe for free to receive monthly, quarterly, and FY email reports with access to the spreadsheets. #️⃣ of Jobs posted - 137 (90% Perm, 4% Fixed-Term, 5% Contract, 1% Part-Time). Top 5 role functions;  ▶ Back End - .Net/C# - 20% ▶ Lead - People (Team Lead, EM, Head of) - 15% ▶ (TIE) Back End - C/C++ and Back End - Python - 9% ▶ Back End - Node.js - 8% Top 5 Sectors;  ▶ Software Development/SaaS - 33% ▶ ITS/Consulting - 19% ▶ Financial Services - 16% ▶ (TIE) Retail/Hospo/Food & Beverage, and Government - 3% Other Insights;  ▶ 47% of jobs were posted by local HQ'd companies (17% uplift from Aug) vs 53% from interstate/overseas. GREAT TO SEE! ▶ We saw a B2B weekly decline in job numbers throughout Sep. Week 1 we saw 41 jobs posted, Week 2 - 36, Week 3 - 31 and Week 4 - 29. That’s a 29% slide. Typically Oct/Nov are strong recruitment months with companies wanting to secure resources by EOY. Let’s see how it plays out. The QLD State Election on the 26th, and the US Election on the 5/11 will have an impact. I know many projects have halted across government departments, and US investors might be more reserved about their spending.  ▶ We saw strong demand again for .Net/C# taking out the #1 spot with a 5% increase. The need for people leaders was apparent again, but we saw a 6% drop in technical leadership jobs, falling out of the top 5.  ▶ It’s interesting to see C/C++ again, driven mainly by companies across the Aviation & Aerospace, Defence and Transport/Logistics/Supply Chain sectors.  ▶ Node & Python made it on the list; I expect them both to be regular features. Node seems to be the go-to language on the back end for most software start-ups/scale-ups (Golang is gaining ground though), and Python a symptom of the big push towards AI/Data. NOTE; ▶ All data is derived solely from LinkedIn.  ▶ I do not include adverts from recruitment agencies.  ▶ Any & all feedback is welcomed 😃 #southeastqldsoftwarejobs #southeastqldsoftwareengineering #softwareengineers #softwaredevelopers #jobsdata

The Data Engineering job market? It’s booming. Here’s what I’m seeing: High Demand. Companies need data to make decisions. That’s driving a big surge in demand for Data Engineers. Organizations everywhere want pros who can build solid data systems. Diverse Opportunities. Data Engineering roles are popping up across sectors—finance, healthcare, tech, retail. There’s a spot for us no matter our interests. Competitive Salaries. With demand up, salaries are following. Companies are offering great compensation to bring in top talent. It shows just how crucial our role is. Evolving Skill Sets. Employers want a mix of skills—SQL, Python, big data tools, plus AWS or Azure. Staying competitive means constant learning and upskilling. Career Growth. The growth path is strong. With experience, we can move up to senior roles, data architecture, or even leadership. It’s a field with real advancement potential. Bottom line? The job market is vibrant and full of opportunities. If you’re thinking about Data Engineering, now’s the time to dive in and make the most of it!

#imhiring #seniorbeackendengineer דרוש.ה Senior Backend Engineer לחברת בינה מלאכותית בת"א   We are developing a platform that collects data from hundreds of different sources and uses machine learning to extract insights for making fast and optimal business decisions. We have assembled an excellent team of leading data engineers, software developers, designers, product managers, and data scientists working towards this goal.  Our offices are in Tel-Aviv and we employ a hybrid work from home model.  Highlights:  Experience with rich, diverse data - one of the truly unique aspects of our domain is the tremendous diversity of data we work with, comprising dozens of sources and spanning thousands of data points and data types  Acceleration - we are just starting our next phase of growth, so whoever joins us now will grow with us  Impact - we are a small team working to impact a multi-billion dollar business  On a typical day you will: Backend Engineering: Develop and maintain scalable backend services using Node.js. Focus on building RESTful APIs, database integration, and implementing effective caching strategies. Expertise in using Docker, Kubernetes, and CI/CD tools like GitHub Actions and Argo is essential.  Cloud Engineering: Deploy and manage applications on cloud platforms such as AWS and Azure. Key areas include ensuring cloud infrastructure security, performance optimization, and scalability. Familiarity with Terraform and monitoring tools like Datadog is crucial.  Data Integration: While less emphasized, proficiency in managing data, particularly in designing ETL processes and handling SQL queries, is important.  Performance Indicators:  Proven ability to lead and deliver complex backend and cloud projects.  Strong collaborative skills across different teams.  Efficient in integrating and managing data in backend and cloud environments.  REQUIREMENTS 4+ years of experience as a Backend Engineer mainly with Node.js & Typescript   4+ years of experience in a cloud environment (AWS, Azure. GCP)  Ability to lead projects on an end-to-end basis  Strong communication skills, problem solving skills  Experience with Nest.js - Advantage  Data Engineering - Advantage  Experience in working on LLM projects – Advantage  Experience with Python - Advantage Proficiency in English is required.    * משרה מס׳ #316373 מיועדת לגברים ונשים כאחד קורות חיים ל: svt.techjobs@gmail.com #backendengineer #nest.js #python #azure

Data Engineer (Python) Worldwide Company: TCIT Salary: $21,600 - $30,000 Type: Remote Employment: Full-time Description: Software Development Leaders in Cloud Modality One of our Missions in Tcit is to discover and strengthen the best talents, so we invite you to be part of our team! Our purpose is to digitally transform differentOrganizations through quality solutions, sustainable and innovative.We work with agile methodologies, we use state -of -the -art technologies and we are software development experts in cloud technologies.Line, a cause management system for judicial courts, a system for certificating monitoring for mining workers, a salary settlement system and other business management solutions in areas such as cargo and winery transport, among many others. In addition, we work in countries like Canada in ambitious projects with our technological partners. Title of Computer Engineer, T.I., computer science or similar. Since 4 years of formal experience as data engineer, with a solid approach in Python (exclusive) Experience proven in cloud services (Azure, AWS, GCP). EXPERIENCE IN THE USE OF TERRAFORM, DOCKER AND KUBERNETES. Advanced knowledge of Azure Functions and APIS development. Experience with SQL databases. Knowledge in web development with Django. Availability to work hybrid (teleworking and office). Ability to work in multidisciplinary teams and effective communication. One of our Missions in TCIT is to discover and enhance the best talents, so we invite you to be part of our team! Our purpose is to digitally transform different organizations through solutions fromquality, sustainable and innovative.We work with agile methodologies, we use state -of -the -art technologies and we are experts in software development in cloud technologies.Of the causes for the judicial courts, a system for monitoring certifications for mining workers, a salary liquidations system and other business management solutions in areas such as cargo and winery transport, among many others. In addition, we work in countries like Canada in ambitious projects with our technological partners. What will you do? You will design, implement and maintain scalable data solutions using Python as the main stack You will develop and manage cloud infrastructures using Terraform, Docker and Kubernetes. You will implement andYou will manage services in Azure Functions and other cloud platforms.. You will collaborate with multifunctional equipment to integrate data solutions with applications developed in Django. You will participate in the lifting of requirements to build, test and generate the product, according to defined times and goals. You will actively participate in the design of the systems, platforms or applicationsof the projects. Apply here: https://lnkd.in/et--ycSk

🚀 Job Hunt Diaries 🚀 The most common challenge I've encountered in my job hunt for a Data Engineer role is the need for "5-10 years of experience" in the domain. While this is understandable, I believe it can sometimes be a bit unfair. When I started my journey as a Software Engineer, I was fortunate enough to have a mentor, Sagar Thakare, who believed that work on paper does not define experience. During my time on his team, he pushed me to learn and implement continuously. I worked on a fantastic stack, developing, deploying, and hosting various software using Jenkins, Docker, Kubernetes, Core Java and Spring Boot. I also worked on automating the SDLC lifecycle. Alongside experienced software engineers, I learned the nitty-gritties of data pipelining, clean coding, library management, and deploying with version control. I dealt with many repositories and even had the opportunity to build my own data pipelines end-to-end, test them, and deploy changes. This work would never have been possible had my mentor not prioritized potential over "work on paper." While this has been a steady blocker for me, Data Engineering is something I'm truly passionate about. All I can do is keep maximizing my efforts: 1. Upskill and keep polishing my skills. 2. Reach out and network. 3. Prepare, prepare, and prepare. The journey continues! 🚀 Lastly: "In the middle of difficulty lies opportunity." - Albert Einstein #JobHunt #DataEngineering #CareerGrowth #Networking #Learning #SoftwareEngineering #Java #Spring #Kubernetes