In the example you provided ("Eu vou sair de férias na quinta"), "na quinta" means "on Thursday". In this case, "quinta" is a short way of saying "quinta-feira", which is the full name of this weekday.
Weekdays are feminine, except for Sunday and Saturday. So you could say "eu vou sair de férias..."
- na segunda: on Monday
- na terça: on Tuesday
- na quarta: on Wednesday
- na quinta: on Thursday
- na sexta: on Friday
- no sábado: on Saturday
- no domingo: on Sunday
This kind of conversation is very common:
-Quando é a festa? (When is the party?)
-Na quinta (On Thursday)
If you want to say "on [Month], 5th", the most common way is "no dia cinco":
Eu vou sair de férias no dia cinco.
It means I'm going on vacations on the next 5th:
- if today is from 1st to 4th, it means 5th of this month
- if today is after 5th, it means the 5th of the next month
You can also say "no dia cinco de [month name]" if you want to specify the month.
As a general rule, you can always say "no dia [number]". So, "no dia vinte e cinco" means "on 25th".
The only exception is the first day of the month: in Brazil, we say "primeiro" (first) instead of "um" (one) - so "no dia primeiro" is "on 1st". And the month can always be omitted, following the rule described above.
Notes:
- as Jacinto commented, in Portugal they say "no dia um". You can also use this in Brazil - and be perfectly understood - but "no dia primeiro" is much more common. I also don't know if this applies to all places in Brazil - it wouldn't be a surprise if "no dia um" is more used in some region, although I'm not aware of a region where this happens.
- it's also possible to omit "no", so instead of saying "Vou sair de férias no dia 5", you could say "Vou sair de férias dia 5" - this is, in my opinion, more informal, and I also suspect that's grammatically incorrect (although very common in conversations).
Saying "no quinto" would sound strange in this context. A situation where you could say "no quinto dia" (on the fifth day):
- if you are planning a trip and wondering what you'll do on the fourth and fifth day of the trip, you could say:
No quarto dia (da viagem) vou fazer X, e no quinto farei Y
On the fourth day (of the trip) I'll do X, on the fifth, I'll do Y
In this case, I could use just "no quinto" (on the fifth), because the word "dia" (day) is implicit - because it was already mentioned before in the sentence.
Note that it refers to the fifth day of the trip, and not necessarily to the 5th of the month.
PS: you can use "na quinta" with a different meaning in situations like this:
I'm trying to do something and it worked just after 5 attempts. You could say:
As primeiras quatro tentativas não deram certo, mas funcionou na quinta.
The first four attempts didn't work, but the fifth did.
It means that it worked in the fifth attempt. In this case, "quinta" isn't the weekday, it's specifying the attempt that worked.