I'm going to outline the process I use to reencode HD-DVDs and BluRay discs. If you don't know how to use these tools don't ask me how to use them as I don't have the time to teach people how to.

For HD DVD titles with VC-1 video and DD+ audio:

You need:
Elecard xmuxer pro or MPEG-2 decoder plugin (elecard MPEG demultiplexer directshow filter) (commercial, easily found)
Sonic Cinemaster decoder package 4.2 (commercial, easily found)
EvoDemux
eac3to.exe (1.9 is latest as of June 29)
virtualdubmod or a graphical avisynth frontend such as AvsP
avisynth 2.5.7
graphedit
MeGUI (preferrably)

Directions:

Load evodemux. Go to options and select "continue with second EVO", "fast EVO checking". Pick a folder to demux into (the folder must already exist). Load the first EVO file of the movie, evodemux will process both evo files. Select the VC-1 video stream from the video tab, select the first EAC3 audio stream (or DTS stream) from the audio tab. Hit "demux".

You get a .MPV file and a .EAC3 file. Convert the EAC3 audio to AC3.

[pre]eac3to source.eac3 destination.ac3[/pre]

eac3to will tell you the length of the audio. Open MeGUI and load its bitrate calculator. Enter the audio length reported by eac3to in here and pick 23.976 for framerate. A frame calculation will come up. Jot this down. Enter 640 for the audio bitrate for track 1, then pick a target filesize (DVD5, DVD9). If you want 1.5 DVD5, calculate the DVD5 bitrate with no audio track, multiply by 1.5, then subtract 640 to get the correct target bitrate. For ALL target sizes, subtract about 5kbps off the bitrate to compensate for x264 overhead and to leave room for muxing subtitles.

Load the MPV file in graphedit. Build a graph of: File source (your MPV file) -> Elecard MPEG Demultiplexer -> Sonic Cinemaster Video Decoder 4.2. Save this graph.

Create the following .avs script and load it in MeGUI's preview or virtualdubmod:

[pre]directshowsource("yourmovie.grf",audio=false,fps=23.976,framecount=XXXXXX)
converttoyv12()[/pre]

Proceed to the "common directions" section below.

For HD DVD titles with h264/AVC video and DD+ audio:

You need:
all tools mentioned above
dgavcindex 0.5a or newer

Demux the video/audio and encode the audio as per above directions for VC-1 discs, calculate bitrate as per VC-1 demuxing method.

Rename the video to .264. Load the video in dgavcindex and create a project file, it will produce a .dga file. NOTE: Some video streams have PAFF frames. dgavcindex will crash or give you an error message if they do. If the disc has PAFF frames, use the graphedit/sonic method described for VC-1 video on the h264 video stream. Note that some streams work with dgavcindex, but the colors are weird / become weird after a certain amount of time. If using dgavcindex, preview it in virtualdub and seek around to make sure it doesn't screw up.

Create the following .avs script and load it in MeGUI's preview or virtualdubmod:

[pre]avcsource("yourmovie.dga",audio=false) # if stream was decodeable with dgavcindex, you do not need a framecount value this way
directshowsource("yourmovie.grf",audio=false,fps=23.976,framecount=XXXXXX) # if stream was not decodeable with dgavcindex
converttoyv12()[/pre]

Proceed to the "common directions" section below.

For BluRay titles with MPEG-2 video and AC3 or LPCM audio:

You need:
dgindex
optional: xport
optional: eac3to.exe (1.8 is latest as of June 3)
virtualdubmod or a graphical avisynth frontend such as AvsP
avisynth 2.5.7
MeGUI (preferrably)

Directions:

If the movie is spread across multiple m2ts files, concatenate them together in the correct order via the copy /b command. copy /b 0001.m2ts+0002.m2ts+0003.m2ts out.m2ts.

Load the m2ts file in dgindex just like a regular transport stream.

Save a d2v project file. Normally the english AC3 track is the first track in the m2ts. If the english track is not the first track (rare, but it happens) you will need to select the audio PID manually with the Set PIDs option, or extract it manually with xport.

If you are planning on extracting the LPCM audio for FLAC or DTS creation, you will need to do so with xport.

[pre]xport -h 0001.m2ts 1 0 1 0 # (replace audio PID with the LPCM PID if it is not 1)[/pre]

Play the AC3 audio and note its length. Open MeGUI and load its bitrate calculator. Enter the audio length in here and pick 23.976 for framerate. A frame calculation will come up. Jot this down. Enter 640 for the audio bitrate for track 1, then pick a target filesize (DVD5, DVD9). If you want 1.5 DVD5, calculate the DVD5 bitrate with no audio track, multiply by 1.5, then subtract 640 to get the correct target bitrate. For ALL target sizes, subtract about 5kbps off the bitrate to compensate for x264 overhead and to leave room for muxing subtitles.

Create the following .avs script and load it in MeGUI's preview or virtualdubmod:

[pre]dgdecode_mpeg2source("yourmovie.d2v")
converttoyv12()[/pre]

Proceed to the "common directions" section below.

For BluRay titles with AVC video and AC3 or LPCM audio:

You need:
dgavcindex 0.5a or newer
optional: Sonic Cinemaster decoder package 4.2 (commercial, easily found)
optional: xport
optional: eac3to.exe (1.8 is latest as of June 3)
virtualdubmod or a graphical avisynth frontend such as AvsP
avisynth 2.5.7
MeGUI (preferrably)

Directions:

If the movie is spread across multiple m2ts files, concatenate them together in the correct order via the copy /b command. copy /b 0001.m2ts+0002.m2ts+0003.m2ts out.m2ts.

Demux the disc using xport. Program stream is 1. Video stream is usually 1. Audio stream is 1 or 2 depending on wheither you want LPCM or AC3 audio. xport will tell you which type of audio it's demuxing when run.

[pre]xport -h 0001.m2ts 1 1 1[/pre]

You will get a video file and an audio file. Rename the video file to .264 and load it in dgavcindex. Preview the video stream. If the video stream looks fine and doesn't have weird artifacting or goofy colors, it is not PAFF and dgavcindex will work fine on it. If you get odd colors or artifacting, you will need to use sonic's decoders as follows: Build a graph in graphedit of file source (your .m2ts) -> sonic HD demuxer -> sonic cinemaster 4.2 video decocder, save the graph. If you are using graphedit/sonic to display the video, you must still extract the audio with xport.

[pre]xport -h 0001.m2ts 1 0 1 0 # (replace audio PID with the LPCM PID if it is not 1)[/pre]

Play the AC3 audio and note its length. Open MeGUI and load its bitrate calculator. Enter the audio length in here and pick 23.976 for framerate. A frame calculation will come up. Jot this down. Enter 640 for the audio bitrate for track 1, then pick a target filesize (DVD5, DVD9). If you want 1.5 DVD5, calculate the DVD5 bitrate with no audio track, multiply by 1.5, then subtract 640 to get the correct target bitrate. For ALL target sizes, subtract about 5kbps off the bitrate to compensate for x264 overhead and to leave room for muxing subtitles.

Create the following avs:

[pre]avcsource("yourmovie.dga",audio=false) # if stream was decodeable with dgavcindex, you do not need a framecount value this way
directshowsource("yourmovie.grf",audio=false,fps=23.976) # if stream was not decodeable with dgavcindex
converttoyv12()[/pre]

Proceed to the "common directions" section below.

--------------------------------------------

COMMON DIRECTIONS

Seek to a bright frame, take a screenshot, load it in paint or photoshop, calculate how many pixels of black borders need to be cropped off all edges. You can only crop in multiples of 2, for quality's sake you are better off overcropping by 1px than undercropping by 1px and leaving a black bar. Add

[pre]crop(w,x,-y,-z)[/pre]

to your .avs. w=left, x=top, y=right, z=bottom

CROP ALL BLACK AND/OR DARK GREY COLORED BARS AROUND ALL EDGES FOR THE LOVE OF GOD.

Resize to the appropriate aspect ratio:

[pre]spline36resize(1280,720) # for video with a source roughly 1920x1080 after cropping
spline36resize(1280,688) # for video with a source roughly 1920x1040
spline36resize(1280,544) # video source 1920x816
spline36resize(1280,536) # video source 1920x800 - pick 536 or 528 depending on which one looks better to you.
spline36resize(1280,528) # video source 1920x800[/pre]

You can sharpen when downsizing to 720p if you like. There's assorted guides on how to sharpen which I won't go into here.

Save the .avs.

Encode with x264. Get x264 here. These are pretty good commandlines which I base many of my x264 encodes on:

[pre]x264.exe --pass 1 --bitrate XXXX --stats ".stats" --bframes 3 --b-pyramid --direct auto
--deblock -3:-3 --subme 1 --analyse none --me dia --threads auto --thread-input --cqmfile "prestige.cfg"
--progress --no-psnr --no-ssim --output NUL "yourmovie.avs"

x264.exe --pass 2 --bitrate XXXX --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip
--bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7
--analyse all --8x8dct --trellis 1 --aq-strength 0.3 --me umh --threads auto --thread-input
--cqmfile "prestige.cfg" --progress --no-psnr --no-ssim --output "yourmovie.mkv" "yourmovie.avs"[/pre]

Use mkvmerge to mux the video and ac3 audio together. That's it.

end common directions

Notes:

- Some HD DVDs have oddly named EVO files which evodemux won't detect. Rename the two feature EVOs to "feature_1.evo" and "feature_2.evo" if this occurs.
- The framecount produced by evodemux is generally off by ~1-2 seconds of frames and sometimes results in extra junk frames at the end of the encoded video. I go with the length of the audio to determine video length.
- The eac3to tool can use Nero's source filters using the -nero switch, if you have Nero installed. These are preferrable for transcoding eac3 to ac3 as they do not apply DRC to the audio.
- The Sonic decoder is known to occasionally produce rainbow frame glitches on a very small number of discs. Thus far three I've seen have done it: Half Baked, Dazed and Confused, The Searchers. There is currently no workaround for this.
- When using the Sonic decoder, the commandline will throw out a lot of strange "ContextException" messages. These are normal and can be safely ignored.
- Please consult the eac3toac3 documentation or THIS THREAD for info on how to transcode LPCM to Flac. It's rather self-explanitory.
- The version of xport I have linked to also demuxes subtitles (presentation graphics) hence the last number in the commandlines being 0. The audio stream is the third digit.
- Here is prestige.cfg

[pre]#Prestige CQM

INTRA4X4_LUMA =
6,7,10,16,
7,7,11,17,
10,11,12,20,
12,13,20,16

INTRA4X4_CHROMAU =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTRA4X4_CHROMAV =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTER4X4_LUMA =
10,13,28,41,
13,14,32,84,
28,32,41,111,
41,46,111,16

INTER4X4_CHROMAU =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTER4X4_CHROMAV =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTRA8X8_LUMA =
9,9,10,10,11,16,21,29,
9,9,10,10,11,16,21,29,
10,10,10,10,11,16,22,31,
10,10,10,11,11,17,23,33,
11,11,11,11,11,19,25,36,
12,12,13,13,14,20,27,40,
15,15,16,16,21,27,31,45,
20,20,21,22,36,40,45,16

INTER8X8_LUMA =
12,13,15,18,20,70,163,255,
13,13,16,18,20,72,170,255,
15,16,17,19,21,81,190,255,
18,18,19,21,23,96,228,255,
20,20,21,23,25,120,255,255,
33,34,36,39,45,32,255,255,
64,66,71,80,164,255,255,255,
151,155,169,192,255,255,255,10[/pre]