I am in the process of setting up a virtualized OPNsense firewall on Proxmox on a Thinkcentre 720q. The proxmox host has 3 network interfaces.
- A dual NIC gigabit card where one interface is for WAN and other for LAN, say eth1 and eth2
- Another interface which came with the PC itself, say eth3
PS: I also have a switch for all my other devices.
After some research, I have understood that
- Passing (pass-through) the NIC to the OPNsense VM is better for performance
- Passing it through removes the interface from the host OS
- If passing is not done correctly, you may lose access to Proxmox.
My questions are
- How do I set eth2 to be the LAN port and also use it connect to proxmox?
- If I use point #1 (eth2 for LAN), how much will the throughput of eth2 be affected? (My ISP provides me symmetrical 320 Mbps link speed)
- If I use point #1, will local traffic (traffic handled by my switch) be affected?
- (Optional/Experimental) Since I have a spare port (eth3), can I use it for special purpose (a dedicated management port which will work even if OPNsense is down)?
- If I use point #4, my switch will have two ethernet connections from the proxmox host. Will this cause loops and kill my network?
You can answer this selectively by mentioning the question number.
If you have a better idea regarding how to setup OPNsense on Proxmox, please share.
Edit #1: Thank you for all your responses! It seems I have to study a lot. Let me answer a few questions
- I am not managing workloads for a dozen of people with strict SLAs. I'm just doing it for my family and myself.
- I understand the point that something as critical as a firewall should have its own hardware. However, I just want to experiment with few VMs on Proxmox. I want to setup Proxmox once and let it be.
- I eventually want to get into VLANs but that is not a priority right now. My future plan is to integrate this with some Omada access points.
- I've added a diagram of what I want to do. Please forgive my crude drawing as it's the best I can do for now.
Please let me know if you want some more information
Edit #2: Thank you for sharing your experience with Proxmox and OPNsense. I'm still reading and re-reading all of your comments to check if I have missed anything.
I have made a small mistake of not ordering the dual NIC + angled riser card before the host arrived, so my host is currently idle. When it arrives, and I manage to set it up, I will make a new post and share what i've learnt.
Thank you again!
I have been running PfSense on Proxmox for ages now.
What I do is the following.
Set your pfSense /OPNsense to start at boot when you power on proxmox.
FYI, you might occasionally run into issues where the NIC "GUID" changes so your VM won't be able to start.
When this happens your pfSense/OPNsense VM won't start so your network will be in a "down state". This means DHCP won't be working either, and any PC that were not assigned a static IP won't be able to access the Proxmox GUI to quickly fix the issue.
You might occasionally need to hook up a temporary router between a PC and your Proxmox host to access the web GUI as a result. At least this is what I do when my outrage is longer then a hour.
Thanks, i may go this route.
I think this is the same issue as a Linux host forgetting where to mount a disk since the UUID was not written in fstab.
But why does the GUID change? Can't it be hard-coded?
Honestly I don't know enough to answer that question fully.
From what I understand PCI addresses (01:00.0) are dynamically assigned by the motherboard's BIOS at boot. Adding or removing PCIe devices, enabling M.2 drives, or adjusting BIOS settings often shifts your device addresses up or down which can prevent a VM from starting up.
Reading online though, there now seems to be a workaround to this issue. I might need to give it a shot on my Proxmox machine.
"PCI ID overrides" is the term in this document to search for.
https://pve.proxmox.com/wiki/PCI(e)_Passthrough
I have one more doubt.
(Nothing setup as of now, just checking all possible angles)
Since, as per your suggestion, I have passed the NIC to OPNsense, so Proxmox won't see the dual NIC card. The onboard ethernet port
eth3is connected to a switch and Proxmox will useeth3.You can see that I have a Ubuntu VM. How will the PC communicate with the VM? (Links E and D are hypothetical. I don't know which one is better given my scenario). Will it be
C -> B -> DorC -> A -> E?Case #1:
C -> B -> DThe switch must differentiate between uplink traffic/proxmox traffic/other physical devices connected to the switch (phone/TV). I prefer this solution because even if OPNsense is down, I can still access the Ubuntu server. But the switch must handle this complexity. I will be getting a managed switch anyway.
Case #2:
C -> A -> EThe interface
eth2, which OPNsense is using, must have some sort of routing table which redirects to linksEorFdepending upon destination IP (Probably implemented using virtual bridge or virtual switch). This is simpler for the switch but if OPNsense is down, i lose direct network access to the VM. I can probably access it via Proxmox web-interface.Can you tell me which one is preferred or which one you would have done?
Sorry for my ramblings.
Lines F and E don't exists in your diagram, all your VMs inside of Proxmox are accessible and sharing port eth3. Except Opensense, as its not bridged to eth3, but instead is assigned to the NIC card you have.
OPNSense inside Proxmox is the only VM that will see the NIC card and be the only VM that uses that NIC with those interfaces.
One interface would be for WAN in like you drew, and the other is the LAN port like on any other router. This LAN port needs to connect to a switch as this is where your OPENsence will communication with the rest of the home network and handout DHCP addresses. It's also how you would reach your OPENsense GUI through a browser. (Outside of managing it within Proxmoxs GUI, accessible on eth3)
If your OpenSense VM goes down your home network won't have a router which means no PCs would be able to communicate as they would have no DHCP addresses, so even if all your communication is "inside of Proxmox" your VM still would not get a DHCP addresses.