Senior
Senior
Senior
Senior
Senior

Senior Python Engineer (AI evaluations platform)

Python
AI
API
Natalia Przybył
IT Recruiter
Location
Remote
Experience
Senior

You will be cooperating with a leading provider of AI evaluation and optimization solutions, trusted by multinational companies to optimize AI agents and detect performance issues in large language models.
In this role, you will cooperate with a leading provider of AI evaluation and optimization solutions, directly contributing to a product that helps multinational enterprises optimize AI agents and detect LLM performance issues. You will be responsabile for designing and implementing complex Reinforcement Learning (RL) environments end-to-end, working at the intersection of research, systems engineering, and product. Collaborating with infrastructure and research teams, you will turn conceptual specifications into working environments with verifiable reward structures, automated verifiers, task generation pipelines, and reproducible simulations. These setups will span from API- and web-based tasks to multi-agent simulations, structured reasoning challenges, and knowledge-work environments used at scale by both researchers and customers.

Due to the client’s time zone, we would appreciate a candidate who can work until 10:00 p.m.

Steps in the recruitment process
  1. HR call (max 10 min.)
  2. Phone call with Technical Manager (max 10 min.)
  3. Tech check with the Acai team (~ 30 min.)
  4. Technical interview(s) with our client (30 - 60 min.)

What will be your responsibilities?

  • Design and implement RL environments that support large-scale agent evaluation and reinforcement learning experiments
  • Build task generation pipelines, dynamic datasets, and scripted environments with controlled complexity and stochasticity
  • Develop verifiers and reward models to automatically score trajectories and evaluate model reasoning
  • Collaborate with infrastructure and systems engineers to ensure environments are scalable, reproducible, and instrumented for detailed telemetry
  • Design APIs and orchestration frameworks for running, resetting, and evaluating agents across environments
  • Partner with research and customer teams to translate open-ended specifications into verifiable, testable systems
  • Optimize environment performance, logging, and reward reproducibility across distributed setups

Requirements

  • Ability to work 2:00 p.m. - 10:00 p.m. (CEST)
  • 5+ years as a Python Engineer
  • Experience with frameworks like FastAPI or Django
  • Skills in designing REST APIs
  • Experience with relational databases (PostgreSQL)
  • Experience with cloud environments like AWS

Nice to have:

  • BS/MS in Computer Science, Mathematics, Statistics, or a related quantitative field
  • Experience with Next.js and Material UI
  • Familiarity with vector and columnar databases

160 – 200 PLN

netto per month - B2B

gross per month - Employment Contract
Natalia Przybył
IT Recruiter

Our benefits

01.
Multisport
As we believe in polish saying "clean body, clean soul", we want to provide you with access to sport facilities that you prefer.
sports
02.
Private healthcare insurance
As you join Acaisoft, you will be granted private insurance. If you have a family or a spouse - we can enable the private insurance for them as well.
health
03.
Internal initiatives
In Acaisoft, we regulary organize internal events such as game or movie nights, bar-hopping or sports.
lightbulb
04.
Top class equipment
Before your first day at work, our team will send you all of the equipment you will need. Our standard equipment includes MacBook Pro 16" with keyboard, mouse, headphones and external monitor(s).
workspace premium

Recruitment process

STEP 1
Application
After you apply, we will review your resume and get back to you. If your experience matches the requirements, we will schedule a quick screening call to get to know you.
create
STEP 2
Interview
After successful screening, you will be invited to interview with one of our delivery managers alongside with software engineer who will ask you technical questions.
question_answer
STEP 3
Decision
After the interviews, we will get back to you as soon as possible with a feedback and, if all goes well, job offer. The feedback process usually takes upmost to 5 business days.
forward_to_inbox

See similar jobs

We are constantly seeking for top talent all over the world. If you did not found the job position that suits your skills and experience, do not hesitate to contact us directly - we will be happy to talk to you about your future with us.

Python Engineer with C++ (AI project)

Senior
Senior
Senior
Senior
Senior
location
Remote
Backend
160 – 200 PLN
+ VAT (B2B) monthly
Python
C++
AI
APPLY

Site Reliability Engineer (AI/LLM & Infra)

Senior
Senior
Senior
Senior
Senior
location
Remote
DevOps
160 – 230 PLN
+ VAT (B2B) monthly
Kubernetes
AI
LLM
AWS
APPLY

Python Engineer

Regular
Regular
Regular
Regular
Regular
location
Pune, India
Python
14$/ hour
+ VAT (B2B) monthly
Python
APPLY

Python-Focused Fullstack (+React)

Senior
Senior
Senior
Senior
Senior
location
Remote
Fullstack
140 - 210 PLN
+ VAT (B2B) monthly
Python
React
AI
APPLY

Senior Machine Learning /AI Engineer (RL)

Senior
Senior
Senior
Senior
Senior
location
Remote
Machine Learning
170 - 230 PLN
+ VAT (B2B) monthly
Python
Langchain
APPLY

React Engineer with Tailwind (Vibe coding)

Senior
Senior
Senior
Senior
Senior
location
remote
Frontend
120 – 170 PLN
+ VAT (B2B) monthly
React
Tailwind
AI
APPLY

Got a question?

Join Acaisoft & let's start our adventure together!
Natalia Przybył
Recruiter
We use our own and third-party cookies. By continuing to navigate, we understand that you accept our cookies policy. More info here

Contact us

Thank you
Your message has been sent.
We will respond to you within 24 hours.
HOME
Oops! Something went wrong while submitting the form.
Since working with Acaisoft, our delivery velocity has increased at least 20 times. In the past months, we’ve done more developments than what had been done in the last four years in our company.
Customer Avatar
Pablo Verano
Head of Product
Scoopr
close