Skip to main content

​
Custom model issues

Q: What are the common issues when deploying custom models? Here are key areas to troubleshoot for custom model deployments:

​
1. Deployment hanging or crashing

Common causes:
  • Missing model files, especially when using Hugging Face models
  • Symlinked files not uploaded correctly
  • Outdated firectl version
Solutions:
  • Download models without symlinks using: 
    huggingface-cli download model_name --local-dir=/path --local-dir-use-symlinks=False
  • Update firectl to the latest version

​
2. LoRA adapters vs full models

  • Compatibility: LoRA adapters work with specific base models.
  • Performance: May experience slightly lower speed with LoRA, but quality should remain similar to the original model.
  • Troubleshooting quality drops:
    • Check model configuration
    • Review conversation template
    • Add echo: true to debug requests

​
3. Performance optimization factors

Consider adjusting the following for improved performance:
  • Accelerator count and accelerator type
  • Long prompt settings to handle complex inputs

​
Autoscaling

Q: What should I expect for deployment and scaling performance?
  • Initial deployment: Should complete within minutes
  • Scaling from zero: Requests to a scaled-to-zero deployment return a 503 error immediately while the system scales up. Your application should implement retry logic with exponential backoff. See Scaling from zero behavior for code examples.
  • Scale-up time: Cold starts typically take 1-3 minutes depending on model size
  • Troubleshooting: If deployment takes over 1 hour, this typically indicates a crash and should be investigated
  • Best practice: If you need instant responses without cold starts, set --min-replica-count 1 or higher to keep replicas always running

​
Performance questions

Q: I have more specific performance questions about improvements For detailed discussions on performance and optimization options:
  • Schedule a consultation directly with our PM, Ray Thai (calendly)
  • Discuss your specific use cases
  • Get personalized recommendations
  • Review advanced configuration options
Note: Monitor costs carefully during the deployment and testing phase, as repeated deployments and tests can quickly consume credits.

​
Additional resources